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ABSTRACT 


The  benefit  of  software  cost  estimation  is  universally  recognized  as  one  of  the  cor¬ 
nerstones  of  effective  software  project  management  and  control  Despite  the  advances  of 
computer-based  estimation  tools,  their  accuracy  remains  largely  inadequate,  and  their 
utility  among  software  development  practitioners  is  limited  Consequently,  the  optimal 
estimation  of  software  cost  remains  an  elusive  goal  of  most  project  managers  Central  to 
this  issue  is  the  nature  of  the  data  on  completed  software  projects  that  are  incorporated 
into  the  organization's  database  of  historical  project  results  This  information  forms  the 
basis  for  both  future  project  estimation  and  ex-post-facto  assessment  of  estimation  mod¬ 
els  Actual  project  results  are  typically  the  data  of  choice  for  both  the  calibration  and 
evaluation  processes,  despite  the  fact  that  these  raw  values  disregard  project  inefficien¬ 
cies  such  as  initial  size  underestimation  This  thesis  challenges  the  notion  that  historical 
project  results  represent  the  preferred  and  most  reliable  benchmarks  for  future  estimation 
purposes.  Computer-based  simulation  is  used  to  test  a  proposed  strategy  which  capital¬ 
izes  on  an  organization's  learning  experiences  by  neutralizing  the  cost  excess  caused  by 
the  initial  undersizing,  and  that  derives  a  posterior  set  of  normalized  effort  and  schedule 
estimation  benchmarks  Analysis  of  the  results  indicates  that  normalization  of  the  data 
leads  to  significantly  improved  project  productivity,  more  optimal  cost  estimates,  and 
provides  the  organization  with  increased  potential  for  future  cost  savings 
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I.  INTRODUCTION 


A.  BACKGROUND 

The  benefit  of  reliable  software  cost  estimation  is  recognized  as  one  of  the  corner¬ 
stones  of  effective  software  project  management  and  control  (Boehm,  1981,  p.30)  Nev¬ 
ertheless,  accurate  estimation  of  software  development  costs  remains  an  elusive  goal  of 
most  project  managers,  despite  the  proliferation  of  software  engineering  economic  analy¬ 
sis  techniques  and  the  availability  of  computer-based  software  project  management  tools 
(Abdel-Hamid.  1990,  p  71). 

Software  development  has  traditionally  been  viewed  as  a  discrete  set  of  software  de¬ 
velopment  life  cycle  (SDLC)  phases,  when  in  fact,  research  findings  point  to  a  dynamic 
environment  characterized  by  continuous  changes  over  time  (Goddard  Space  Flight  Cen¬ 
ter,  I  WO)  Consequently,  problems  inherent  with  the  estimation  process  itself,  normally 
positioned  at  the  beginning  of  the  SDLC,  have  generally  limited  the  utility  of  estimation 
tools  based  on  this  traditional  view  of  software  development 

Without  the  benefit  of  full  knowledge  of  a  project's  ultimate  scope  and  definition  at 
the  time  olTmtial  cost  estimation,  an  estimation  model  must  possess  the  capability  to  re¬ 
spond  to  influencing  factors  which  unfold  as  the  project  progresses  through  the  SDLC 
Abdel-Hamid  states  that  ".  estimation  should  be  a  continuous  process  enhanced  through 
constant  updates  of  feedback  data  collected  from  project  monitoring  and  control  activi¬ 
ties..."  He  argues  that  continuous  estimation  models  must  support  the  full  range  of 
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estimation  activities  regularly  encountered  in  the  SDLC,  adaptive  (accommodate  new  or¬ 
ganizational  realities),  corrective  (correct  initial  faulty  assumptions)  and  perfective  (post¬ 
mortems  to  perfect  project  statistics).  In  so  doing,  it  is  imperative  that  the  model  also 
possess  the  capability  to  capture  management-system  dynamics  --  project  managers'  reac¬ 
tions  to  real-world  events  as  they  unfold.  (Abdel-Hamid,  1993  pp  20-21 ) 

Despite  the  improvements  realized  with  the  introduction  of  genuine  continuous  esti¬ 
mation  moods,  their  accuracy  remains  largely  inadequate  Central  to  this  issue  is  the  na¬ 
ture  of  the  data  on  completed  software  projects  that  are  incorporated  into  the 
organization's  database  of  historical  project  results.  This  archived  information  subse¬ 
quently  forms  the  basis  for  both  future  project  estimation  and  ex-post-facto  evaluation  of 
software  cost  estimation  models.  Quite  simply,  this  data  is  used  to  produce  the  organiza¬ 
tion’s  "best  guess"  of  what  a  project  of  similar  size  and  scope  should  require,  in  terms  of 
development  effort  and  schedule,  if  encountered  in  the  future.  In  addition,  it  is  these  data 
values  upon  which  estimation  tool  calibration,  or  fine-tuning  to  produce  more  accurate 
estimates  which  reflect  the  organization's  unique  software  development  environment,  is 
based 

Raw  project  values,  which  represent  actual  results,  are  the  conventional  "data  of 
choice"  for  both  the  estimation  and  calibration  processes.  While  raw  data,  indeed,  reflect 
actual  results,  they  may  certainly  not  reflect  optimum  results,  particularly  in  the  case  of  a 
problematic  project  Inefficiencies  such  as  initial  size  underestimation,  plague  many,  if 
not  all  software  development  projects,  and  are  manifested  in  varying  degrees  of  cost 


overruns  and  schedule  slippages.  As  such,  direct  incorporation  of  raw  values  into  the  his¬ 
torical  database  tends  to  discount  the  impact  of  these  inefficiencies  on  project  results.  In¬ 
stead,  it  merely  archives  this  flawed  information  for  future  ( misapplication,  and 
perpetuates  the  cycle  of  inefficiency  and  imprecise  estimation. 

In  response,  Abdel-Hamid  has  proposed  a  strategy  which  "...capitalizes  on  an  organi¬ 
zation's  learning  experiences,  by  wringing  out  the  cost  excess  caused  by  the  initial  under¬ 
sizing  and  that  derives  a  posterior  set  of  normalized  cost  and  schedule  estimation 
benchmarks."  (Abdel-Hamid,  1993,  p.  28)  These  normalized  values  are  representative 
of  a  perfectly-sized  software  project,  and  consequently  should  provide  the  organization 
with  a  more  efficient  benchmark  for  future  project  estimation  and  planning,  and  in  retro¬ 
spect,  evaluating  how  well  project  resources  were  used. 

B.  PURPOSE  OF  RESEARCH 

This  research  challenges  the  notion  that  raw  historical  values  represent  the  preferred 
benchmark  for  calibrating  software  cost  estimation  models  Computer-based  simulation 
is  used  to  model  the  behavior  of  a  number  of  synthetic  project  profiles  to  test  the  assump¬ 
tions  of  both  the  conventional  and  normalized  strategies  for  software  estimation  model 
calibration.  Various  experimental  conditions  are  imposed  on  subsequent  experiments  to 
compare  project  results  and  identify  causal  relationships  in  an  effort  to  substantiate  the 
research  claims 
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C.  THE  RESEARCH  QUESTION 

The  primary  research  question  of  this  thesis  is  to  determine  if  there  is  long-term  bene¬ 
fit  in  using  normalized  software  project  cost  values  vice  raw  historical  data  as  the  bench¬ 
mark  for  calibrating  software  estimation  models. 

D.  SCOPE  OF  RESEARCH 

The  scope  of  this  research  includes  the  design,  execution  and  analysis  of  a  computer- 
simulated,  multiple-project  experiment,  and  comparing  the  results  of  two  competing  soft¬ 
ware  estimation  calibration  strategies,  in  order  to  answer  the  research  question.  Its  scope 
does  not  extend  beyond  the  research  laboratory,  and  there  are  no  immediate  plans  for 
replicating  this  experiment  in  a  real-world  environment. 

E.  THESIS  ORGANIZATION 

Chapter  II  offers  a  statement  of  the  experiment’s  objectives  and  a  comprehensive  de¬ 
scription  of  the  experimentation  tools,  to  include  the  Constructive  COst  MOdel  of  Soft¬ 
ware  Cost  estimation  (COCOMO)  and  the  System  Dynamics  (SD)  Model  of  Software 
Project  Development  In  addition,  Chapter  II  presents  the  experimental  design,  where  the 
hypothetical  projects,  project  profiles  and  influencing  factors  and  assumptions  are  de¬ 
fined  in  detail  A  key  element  of  Chapter  II  is  a  discussion  of  the  competing  software  es¬ 
timation  model  calibration  strategies  which  form  the  basis  of  this  research.  Chapter  III 
describes  the  experimental  setting  and  related  tasks,  and  elaborates  on  exercise  organiza¬ 
tion,  methodology  and  conduct  In  addition,  the  dependent  measures  which  represent  key 
exercise  metrics,  are  defined  as  they  relate  to  analyzing  and  comparing  exercise  results 
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Chapter  IV  presents  the  results  of  the  various  experiments  and  offers  insight  and  analysis 
of  the  research  findings  Chapter  V  summarizes  the  findings  of  the  previous  chapters, 
discusses  the  implications  of  this  study  ,  and  proposes  related  opportunities  and  directions 
for  future  research 
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II.  METHOD  AND  PREPARATION 

A.  EXPERIMENTAL  OBJECTIVE 

This  experiment  will  use  a  system  dynamics  model  of  software  development  to  simu¬ 
late  the  development  of  a  set  of  30  projects  in  a  software  organization,  conductec  iv 
over  an  approximate  12-year  period  The  simulated  results  will  be  incorporated  into  an 
organizational  data  base  and  used  as  the  basis  for  both  subsequent  project  estimation  and 
calibration  of  the  estimation  tool.  Two  scenarios  will  be  evaluated  the  conventional 
method  of  calibration  using  raw  historical  data  and  an  alternative  calibration  method  us¬ 
ing  "normalized"  metrics 

B.  EXPERIMENTATION  TOOLS 

1.  Constructive  Cost  Model  (COCOMO) 

The  Constructive  COst  MOdel,  or  COCOMO,  was  developed  by  Barry  Boehm,  and 
is  a  widely-accepted  algorithmic  model  which  is  used  to  determine  initial  software 
development  effort  and  schedule  estimates  As  a  result  of  model  refinement  since  its 
introduction,  three  model  versions  and  three  software  development  modes  have  evolved 
The  three  versions  include  Basic,  Intermediate  and  Detailed  COCOMO,  each  of 
increasing  detail  and  accuracy.  Organic,  Semidetached,  and  Embedded  software 
development  modes  have  been  defined  to  accommodate  the  broad  spectrum  of  project 
size,  specificity,  and  risk  encountered  in  the  software  development  environment. 
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Basic  COCOMO  is  the  simplest  version  of  the  model,  and  is  effective  for  rough  order 
of  magnitude  estimates  of  software  cost  However,  Boehm  cautions, "...  its  accuracy  is 
necessarily  limited  because  of  its  lack  of  factors  to  account  for  differences  in  hardware 
constraints,  personnel  quality  and  experience,  use  of  modem  tools  and  techniques,  and 
other  project  attributes  known  to  have  a  significant  influence  on  software  costs  " 
(Boehm,  1981,  p.  58)  With  Basic  COCOMO,  estimates  of  effort  are  generated  using  only 
a  single  predictor  variable,  namely  the  number  of  delivered  source  instructions  (DSI) 
developed  by  the  project 

Intermediate  COCOMO  improves  upon  the  Basic  version  by  incorporating  an 
additional  1 5  predictor  variables,  or  cost  driver  attributes,  which  are  carefully  identified, 
weighted  and  introduced  in  order  to  offset  much  of  the  cost  variation  found  in  Basic 
COCOMO  The  15  cost  drivers  are  subdivided  into  four  categor.es:  software  product 
attributes,  computer  attributes,  personnel  attributes,  and  project  attributes.  Each  cost 
driver  has  an  associated  effort  multiplier  which  is  applied  to  the  nominal  development 
effort  to  obtain  a  more  accurate  estimate  Boehm  contends  that  the  level  of  accuracy 
achieved  with  Intermediate  COCOMO  "...  is  representative  of  the  current  state  of  the  art 
in  software  cost  models."  (Boehm,  1984,  p.  16) 

Detailed  COCOMO  provides  the  highest  level  of  estimation  accuracy  by  providing 
even  more  detail  as  model  input.  This  is  accomplished  by  employing  a  three-level 
hierarchical  decomposition  of  the  software  product  whose  cost  is  to  be  estimated  In 
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addition,  phase-sensitive  effort  multipliers  are  used  to  accurately  reflect  the  effect  of  the 
cost  drivers  on  the  phase  distribution  of  effort  (Boehm,  1981,  pp.  347-348) 

The  three  COCOMO  modes  of  software  development  were  defined  as  a  result  of 
research  findings  suggesting  that  software  products  of  the  same  size  often  require  varying 
degrees  of  effort  and  development  time  Consequently,  each  of  the  COCOMO  software 
development  mode's  effort  and  schedule  equations  will  yield  significantly  different  cost 
estimates  Hence,  precise  identification  of  the  applicable  mode,  by  means  of  its 
distinguishing  features,  is  critical  in  order  to  prevent  estimation  inaccuracies. 

The  organic  mode  represents  projects  that  are  relatively  small  in  size,  developed  by 
small  software  teams  in  a  generally  stable  development  environment  Experience  levels 
are  high,  while  schedule  and  performance  pressures  are  generally  lower 

The  semidetached  mode  represents  the  middle  ground  between  the  organic  and 
embedded  modes  Flexibility  of  approach  is  a  trademark  of  the  semidetached  mode,  as 
intermediate  levels  of  project  characteristics  and  a  blend  of  organic  and  embedded  mode 
characteristics  may  be  encountered  in  the  same  project 

Finally,  the  embedded  mode  represents  a  project  that  must  operate  within  tight 
constraints.  Requirements  and  interface  specifications  are  generally  inflexible,  and  can 
dictate  a  considerable  need  for  innovative  architectures,  algorithms  or  functionalities 
(Boehm,  1981,  p  81 ) 

In  this  series  of  experiments,  the  Basic  COCOMO  version  will  be  utilized  as  the 
software  estimation  model  While  intermediate  COCOMO  estimates  have  proven  clearly 
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superior,  the  rudimentary  nature  of  the  Basic  COCOMO  (only  size  input  -  no  cost  driver 
attributes)  facilitates  evaluation  of  model  characteristics  in  conjunction  with  the  SD 
simulator.  Likewise,  the  organic  software  development  mode  complements  the  choice  of 
Basic  COCOMO,  and  assumes  a  stable  baseline  software  development  environment  in 

which  the  experiments  can  be  conducted. 

2.  A  Dynamic  Simulation  Model  of  Software  Development 

Research  has  underscored  the  im practicalities  of  controlled  experimentation  in  the 

software  engineering  field  as  being  excessively  costly  and  time-consuming  (Myers, 
1978).  Simulation  modeling  provides  a  flexible  and  ideal  environment  in  which 
competing  assumptions  and  conditions  may  be  tested.  Unlike  real  systems,  the  effects  of 
variable  manipulation  on  internal  system  interactions  can  be  isolated  and  more  carefully 
studied.  Consequently,  for  purposes  of  this  experiment,  simulation  modeling  was  chosen 
as  the  experimental  method  by  which  the  research  question  would  be  answered. 

The  System  Dynamics  (SD)  Model  of  Software  Project  Development,  by 
Abdel-Hamid  and  Madnick,  is  a  comprehensive,  highly-detailed,  quantitative  simulation 
model  which  captures  management-system  dynamics  and  provides  a  continuous 
simulation  capability.  Based  on  the  feedback  principles  of  system  dynamics,  the  model 
focuses  on  four  interconnected  subsystems,  which  integrate  managerial  decision-making 
activities  (e  g.,  scheduling,  productivity,  and  staffing)  with  the  physical  production  of  the 
software  product  (e  g.,  design,  coding,  reviewing,  and  testing).  The  four  subsystems  are 
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human  resource  management,  software  production,  controlling,  and  planning 
(Abdel -Ham id,  1993,  p  24) 

The  purpose  of  the  SD  simulator  is  to  serve  as  a  laboratory  vehicle  for  conducting 
experimentation  into  the  dynamics  of  software  development  As  such,  it  provides  a 
much-needed  means  by  which  the  managerial  side  of  the  software  development  process 
might  be  more  carefully  examined  and,  hopefully,  better  understood  By  design,  the 
model  does  not  deliver  point  predictions,  but  rather  seeks  to  provide  a  general 
understanding  of  the  nature  of  the  dynamic  behavior  of  a  project  An  important 
functionality  of  the  model  is  the  ability  to  perform  sensitivity  analysis,  or  "what-if' 
experiments,  in  order  to  develop  a  more  complete  understanding  of  the  interrelationships 
of  software  development  variables  and  identification  of  causal  relationships 

The  model  has  been  designed  for  use  on  medium  sized,  organic  type  software 
projects  (i.e.,  projects  that  are  10,000  to  250,000  lines  of  code  and  conducted  in  familiar, 
in-house  development  environments)  (Stephan,  1 992,  p.  13).  For  a  detailed  discussion  of 
the  model's  actual  structure,  formulation  and  validation,  see  Abdel-Hamid  and  Madnick 
(1989  and  1991) 

C.  EXPERIMENTAL  DESIGN 

1.  Definition  of  Experimental  Projects 

Five  hypothetical  software  development  projects,  of  varying  representative  sizes, 
were  initially  defined  and  serialized  as  projects  one  through  five.  Their  size  was 
established  in  terms  of  thousands  of  delivered  source  instructions  (KDS1)  to  match  both 


the  COCOMO  and  SD  simulator  input  parameters.  Table  1  presents  project  senals  and 
their  respective  sizes,  which  remain  fixed  throughout  all  experiments. 


Project  Serial 

Actual  Size  (KDSI) 

1 

40 

2 

50 

3 

60 

4 

70 

5 

80 

Table  1.  Experimental  Projects  and  Sizes 


2.  Underestimation  of  Project  Size 

Boehm  states,  "The  software  undersizing  problem  is  our  most  critical  road  block  to 
accurate  software  cost  estimation "  He  cites  three  mam  reasons  for  this  perplexing 
phenomenon.  First,  people's  optimistic  and  accommodating  nature  drive  them  to  say 
what  others  want  to  hear  High  estimates  are  fuel  for  confrontation,  whereas  everyone  is 
happy  with  small,  easy  software.  The  second  reason  involves  incomplete  recall  of  the 
large  amount  of  support  software  that  must  be  developed  as  part  of  a  project  —  there  is 
generally  a  stronger  recollection  of  the  size  and  effort  required  for  the  much  smaller,  but 
more  visible,  operational  software  The  third  reason  is  related  to  the  incomplete  recall 
issue.  Unfamiliarity  with  the  full  scope  of  the  software  project  causes  people  to  overlook 
the  more  obscure  software  products  (and  obscure  portions  of  each  product)  which  need  to 
be  developed.  There  are  no  quick  fixes  to  the  pervasive  undersizing  problem  other  than 
to  understand  the  sources  of  the  problem,  and  apply  that  understanding  to  software  sizing 
activities.  (Boehm,  1981,  pp.  320-323) 
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A  study  of  the  impact  of  undersizing  on  software  estimation  forms  the  focus  of  much 
of  this  experiment  Consequently,  underestimation  levels,  expressed  as  a  percentage  of 
actual  project  size,  are  applied  to  the  individual  project  serials  in  accordance  with  the 
experimental  project  profile,  which  is  defined  m  a  subsequent  section  of  this  report 
Underestimation  levels  are  defined  and  presented  in  Table  2 


Level 

Underestimation  (%) 

1 

10 

2 

20 

3 

30 

4 

40 

5 

50 

Table  2.  Project  Size  Underestimation  Levels 


Undersizing  has  a  direct  effect  on  both  the  software  cost  model  (COCOMO)  and  the 
simulation  model  (SD  simulator)  results.  Quite  simply,  a  too-small  sizing  estimate 
invariably  results  in  a  too-small  cost  estimate  For  example,  a  50  KDSI  project, 
undersized  by  20  percent,  results  in  a  Basic  COCOMO  estimation  identical  to  that  of  an 

accurately-sized  40  KDSI  project. 

3.  Development  of  Project  Profiles 

The  experiment  seeks  to  model  and  analyze  the  software  development  activities  of  a 
hypothetical  organization  over  time  In  developing  a  project  profile  for  the  organization, 
particular  attention  was  paid  to  a  number  of  conditions  within  the  organization  that 
would  accomplish  exercise  objectives,  while  maintaining  a  reasonable  degree  of  realism 
with  respect  to  the  functioning  of  an  actual  software  development  organization 
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a.  Project  Teams 

Five  hypothetical  software  development  teams  are  constructively  assembled  As 
teams,  they  will  be  assigned  to  one  of  the  project  serials  -  one  team  for  each  project 
serial.  There  was  no  consideration  given  to  team  make-up  in  assembling  the  teams. 
Although  disregard  for  the  effects  of  personnel  attributes  on  team  performance  represents 
an  exercise  artificiality,  the  assumption  of  essentially  "homogeneous"  project  teams 

facilitates  unbiased  interpretation  of  the  exercise  results. 

b.  Project  Cycles 

In  order  to  investigate  the  long-term  impact  of  calibration  strategies  on  software 
cost  estimation,  follow-on  projects  to  the  five  project  serials  already  defined  is  required 
Consequently,  the  concept  of  a  project  cycle  is  introduced.  A  project  cycle  is  defined  as 
that  period  of  time  required  for  each  of  the  five  individual  project  senals  to  be 
completed  The  first  iteration  of  this  scheme  is  referred  to  as  "Project  Cycle  One", 
whereas  subsequent  iterations  are  labeled  "Project  Cycle  Two",  "Project  Cycle  Three”, 
etc.  For  purposes  of  this  experiment,  organizational  software  development  activities  will 

span  six  project  cycles. 

c.  Initial  Project  Team  Assignments 

With  teams  assembled,  and  projects  and  project  cycles  defined,  the  next  step  is  to 
determine  a  strategy  for  project  assignment.  Here  the  assumption  is  that  all  five  software 
development  teams  will  commence  work  on  the  five  project  serials  concurrently,  at  time 
zero.  For  simplicity,  and  to  provide  a  convenient  project  profile  starting  point. 
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assignment  of  projects  in  project  O'de  one  matches  team  one  with  project  one,  team  two 
with  project  two,  etc.  Table  3  outlines  cycle  one  project  assignments 


Project  Cycle  One 

Project  Team 

Project  Assignment 

One 

1 

Two 

2 

Three 

3 

Four 

4 

Five 

5 

Table  3.  Cycle  One:  Team  and  Project  Assignments 


d.  Allocation  of  Undersizing  Factors 

In  order  to  examine  the  effects  of  undersizing  on  projects  of  varying  size,  the 
previousl\ -defined  size  underestimation  levels  (Table  2)  must  be  allocated  in  a  random 
manner  across  all  projects  For  project  cycle  one,  this  was  accomplished  by  using  a  table 
of  numbers  generated  by  a  random  process.  Table  4  is  such  a  table  and  is  used  in  the 
experiment  By  arbitrarily  selecting  the  intersection  of  any  row  and  column  as  the 
starting  point,  a  list  of  five  numbers  is  systematically  drawn  by  moving  either  to  the  left 
or  right,  or  upward  or  downward  from  this  starting  point  until  one  of  the  underestimation 
level  values  is  encountered  This  number  is  recorded  in  the  list,  and  the  movement 
continues  until  a  second  number  within  the  allowable  range  (one  through  five)  is 
encountered  After  this  second  value  is  recorded  in  the  list,  the  process  repeats  three 
more  times  until  the  randomized  list  of  five  numbers  is  complete.  For  example, 
underestimation  levels  are  allocated  for  project  cycle  one  by  choosing  row  5,  column  13 


Table  4  Table  of  Random  Numbers  After  Ref.  (Roscoe,  1975,  p  410) 
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Column  Number 


(Table  4)  as  the  starting  point  and  moving  across  the  row  to  the  right.  The  following 
randomized  list  is  generated:  4-2-3-5-1.  These  numerical  values,  corresponding  to 
underestimation  levels,  are  allocated  to  cycle  one  projects  as  shown  in  Table  5 


Project  Cycle  One 

Project  # 

Undersizing  Level 

1 

4 

2 

2 

3 

3 

4 

5 

5 

1 

Table  5.  Cycle  One:  Projects  and  Undersizing  Levels 


For  project  cycles  two  through  six,  undersizing  levels  are  allocated  in  accordance 
with  the  Latin  Square  Design  (Daniel  and  Terrell,  1975,  pp.  209-215).  Once  the 
cycle-one  undersizing  levels  are  determined  and  allocated  to  the  five  project  serials  in 
ascending  project-size  order,  Latin  Square  imposes  a  one-position  downward  shift  of  row 
values  to  produce  the  undersizing  allocation  for  cycle  two.  The  procedure  is  repeated 
through  the  six  project  cycles,  which  results  in  cycle-six  undersizing  levels  identical  to 
those  in  cycle  one.  Table  6  presents  the  undersizing  allocation  for  all  projects  across  all 
project  cycles.  This  allocation  plan  is  fixed,  and  is  used  for  all  experiments  where 
software  size  underestimation  is  assumed 
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Project  # 

KDSI 

Project  Cycle 

1 

_ 

2 

_L_i 

4 

_ 

5 

6 

Underestimation  Level 

1 

40 

4 

1 

5 

3 

2 

4 

2 

50 

2 

4 

1 

5 

3 

2 

3 

60 

3 

2 

4 

1 

5 

3 

4 

70 

5 

3 

2 

4 

1 

5 

5 

80 

1 

5 

3 

2 

4 

1 

Table  6.  Project  Undersizing  Allocation 


e  Project  Team  Assignments  in  Cycles  Two  through  Six 

In  developing  the  project  profile,  it  was  decided  that  when  a  project  team 

completed  their  assigned  project  in  cycle  one,  they  would  immediately  be  assigned  a  new 
project  and  commence  work  in  cycle  two.  That  is,  the  team  that  finishes  their  cycle-one 
project  first,  is  assigned  the  first  available  project  in  cycle  two  The  second  team  to 
finish  cycle  one  gets  the  next  available  project  in  cycle  two,  and  so  on,  until  all  five 
teams  "arrive"  in  project  cycle  two  Subsequent  project  assignments  are  determined  in 
the  same  manner  through  project  cycle  five. 

The  sequence  of  next-available  projects  for  project  cycles  two  through  five  are 
randomly  assigned  Their  project  assignment  orders  are  determined  by  employing  the 
same  randomization  techniques  described  in  the  previous  section,  but  with  different 
starting  coordinates  and  directions  of  movement  for  generating  the  randomized  list  for 
each  cycle. 
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To  facilitate  comparative  analysis  of  results  with  cycle  one  projects,  cycle  six 
team  assignments  replicate  their  initial  project  assignments.  Table  7  defines  the 


next-available  project  scheme  for  all  six  project  cycles. 


Order  of 
Project 
Completion 
in  Present 
Cycle 

Project  Cycle 

2 

3 

4 

5 

6 

Next-Avalibale  Project 

1 

2 

3 

1 

5 

1 

2 

1 

4 

4 

4 

2 

3 

3 

1 

5 

2 

3 

4 

5 

5 

3 

1 

4 

5 

4 

2 

2 

3 

5 

Table  7.  Next- Available  Project  Schedule 


f.  Finalized  Experimental  Project  Profile 

The  final  project  profile,  which  incorporates  next-available  project  assignments 
and  their  respective  undersizing  levels,  is  presented  in  Table  8.  All  experiments  follow 
this  project-order  and  undersizing  arrangement  (when  applicable).  While  project  team 
assignments  in  other  than  the  initial  project  cycle  may  vary  under  different  exercise 
scenarios,  depending  on  calculated  total  development  schedule  values,  the  follow-on 
project  order  and  underestimation  levels  of  Table  8  remain  fixed  in  all  cases  Figure  1 
displays  a  representative  Total  Development  Schedule  for  ail  five  project  teams  over  six 
project  cycles,  applying  the  experimental  project  profile 
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4.  Learning 

The  effects  of  "learning"  on  software  estimation  and  productivity  are  an  important 
element  of  this  research.  It  is  reasonable  to  assume  that  the  effect  of  experience  and 
increases  in  project  familiarity  should  be  reflected  in  higher  productivity  In  an  attempt 
to  model  the  rate  of  learning  improvement,  a  plan  involving  the  incremental  increase  of  a 
related  SD  simulator  input  variable  was  developed. 

In  the  SD  model,  nominal  productivity  is  defined  as  one  task  per  man-dav  A  task  is 
any  arbitrary  unit  by  which  a  software  project  may  be  measured  (Abdel-Hamid  and 
Madnick.  1991.  p  80).  In  our  experimentation  vehicle,  a  "task"  is  defined  in  terms  of  a 
discrete  number  of  Delivered  Source  Instructions,  hence  the  SD  input  parameter 
Delivered  Si  nin  e  Instructions  per  Task  (DSIPTK).  Consequently,  an  appropriate 
increase  in  DSIPTK  over  the  nominal  simulator  value  as  projects  are  developed,  can 
effectivclv  model  the  'learning  curve'  effect  we  are  searching  for 

For  purposes  of  this  experiment,  we  assume  that  "learning"  is  reflected  in  a 
10-percent  annual  increase  in  DSIPTK.  While  total  project  development  schedules 
obviously  vary,  an  18  to  24-month  timeframe  represents  a  reasonable  estimate  of 
duration  for  the  hypothetical  projects  as  defined.  Consequently,  a  20-percent  increase  in 
DSIPTK  was  applied  to  each  project  cycle  beginning  with  project  cycle  two.  This  value 
is  consistent  with  research  findings  and  industry  experiences  (Aron,  1976).  Hence,  the 
learning  scenario  is  defined  as  an  incremental  increase  of  DSIPTK  from  100  percent  of 


nominal  value  to  200  percent  of  the  nominal  SD  simulator  value  over  the  six  project 
cycles.  Table  9  demonstrates  how  the  learning  scenario  was  applied. 


Project  Cycle 

DSIPTK:  Percent  of 
Nominal  Value 

1 

100% 

2 

120% 

3 

140% 

4 

160% 

5 

180% 

6 

200% 

Table  9.  Learning  Scenario 


5.  Conventional  COCOMO  Calibration  Strategy 

"Calibration"  is  one  method  by  which  an  organization  may  tailor  a  software 
cost-estimation  tool  to  more  accurately  reflect  its  unique  software  development 
experiences.  Boehm  asserts  that  calibration  of  COCOMO  may  be  necessary,  for  various 
reasons,  to  provide  an  organization  with  the  best  estimation  accuracy  "fit"  He  offers  a 
technique  for  calibrating  the  constant  term  in  the  COCOMO  nominal  effort  equation,  and 
this  procedure  will  be  replicated  as  part  of  the  experiment,  and  throughout  the  thesis  will 
be  referred  to  as  the  "conventional"  calibration  strategy. 

Having  selected  the  Basic  COCOMO  model  and  the  organic  mode  as  the  most 
appropriate  software  development  mode  for  our  hypothetical  organization,  the  calibration 
methodology  is  straightforward.  Table  10  presents  the  Basic  COCOMO  effort  and 
schedule  equations  for  the  organic  mode.  A  few  terms  require  definition  in 
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understanding  these  equations.  Under  the  Effort  column,  "MM"  refers  to  the  number  of 
man-months  estimated  for  the  software  development  phase.  One  man-month  is  equal  to 
152  hours  of  working  time.  Under  Schedule ,  "TDEV"  is  the  number  of  estimated 
months  for  software  development 


Mode 

Effort 

Schedule 

Organic 

MM  =  2.4  (KDSI)"’< 

TDEV  =  2.5  (MM)0'* 

Table  10.  Basic  COCOMO  Effort  and  Schedule  Equations  (Organic  Mode) 

The  constant  term  in  the  effort  equation  above  (2.4)  is  the  value  which  is  calibrated. 
Because  of  the  absence  of  cost  driver  attributes  in  Basic  COCOMO,  the  optimal 
coefficient  may  be  calculated  using  the  following  equation: 

p,  Xt=  1  MMi(actual)*Qi 

C  = - - I -  (2.0 

In  the  above  equation,  MM,(  actual)  is  the  actual  development  effort  of  the  software 
project  In  our  experiment,  this  value  is  generated  by  the  SD  simulator,  based  on  input 
values  which  include  the  Basic  COCOMO  effort  and  schedule  estimates.  The  variable  Q, 
for  organic  mode  re-calibration,  is  defined  as  the  actual  size  of  the  project  (KDSI(actual)) 
raised  to  the  power  1.05.  Having  determined  these  values,  the  calibration  process 
continues  by  multiplying  MM, (actual)  times  Q,  for  each  project.  The  summation  of  this 
product  is  determined  for  the  number  of  projects  being  factored  in  to  the  re-calibration 
(n).  This  value  forms  the  numerator  of  the  re-calibration  equation.  The  denominator  is 
calculated  by  first  squaring  each  Q,  value,  then  summing  these  values.  The  resultant 
coefficient  represents  the  denved  optimal  constant  term  and  replaces  the  organic 
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COCOMO  coefficient  value  of  2.4  for  estimation  of  the  next  senes  of  (n)  projects 
Chapter  IV  provtdes  additional  clarification  of  the  calibration  methodology  using 
exercise  data. 

6.  Alternative  "Normalization"  Calibration  Strategy 

Boehm  commented  on  a  comparative  analysis  of  software  cost  models,  that  "  Not 
too  surprisingly,  the  best  results  were  generally  obtained  using  models  with  calibration 
coefficients  against  data  sets  with  few  points...  "  (Boehm,  1984,  p  18).  A  similar 
analysis  of  the  validity  of  the  assumptions  upon  which  calibration  strategies  are  based, 
and  their  impact  on  software  estimation  model  performance  has  received  considerably 
less  attention 

Basic  COCOMO  embraces  the  assumption  that  historical  project  results  represent  the 
preferred  and  most  reliable  benchmarks  for  future  estimation  purposes  This  experiment 
challenges  that  notion,  and  seeks  to  validate  the  work  of  Abdel-Hamid  by  using  the  SD 
model  as  an  experimental  vehicle  to  demonstrate  why  this  assumption  is  flawed 
(Abdel-Hamid,  1990,  p  79). 

Using  data  from  a  real  software  project  conducted  by  NASA,  Abdel-Hamid 
conducted  two  experiments  as  part  of  SD  model  validation.  The  first  experiment 
investigated  one  of  two  fundamental  assumptions  upon  which  conventional  calibration 
strategies  are  based.  That  is,  a  project's  final  results  are  independent  of  its  initial 
estimation  values.  His  research  findings  indicate  that  different  estimates  do,  indeed, 
create  different  projects  He  reported  that  initial  project  effort  and  schedule  estimates 
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significantly  influence  work  force  level  decisions,  productivity,  work  intensity,  and 
communication  and  training  overheads  Clearly,  acceptance  of  these  findings  leads  to 
rejection  of  the  convention  that  actual  project  results  provide  the  best  information  for 
future  estimation  activities. 

Abdel-Hamid's  second  experiment  sought  to  further  refute  the  notion  that  raw 
historical  project  values  should  be  the  "data  of  choice"  for  both  the  calibration  and 
ex-post-facto  evaluation  of  estimation  tools.  Again,  using  the  NASA  data,  he  reported 
how  the  initial  35-percent  size  underestimation  lead  to  a  corresponding  underestimate  of 
project  effort  and  duration.  He  observed  how  learning,  in  the  form  of  increased  project 
familiarity  and  experience,  lead  to  the  discovery  of  overlooked  tasks,  which  in  turn 
resulted  in  a  dramatic  "staff  explosion"  late  in  the  development  cycle,  in  order  to  meet  a 
rigid  deadline  At  this  point,  the  representativeness  of  NASA’s  actual  project  cost  as  the 
basis  for  future  effort  estimation  becomes  suspect  due  to  the  problematic  nature  of  the 
project.  A  new  project  of  similar  size  and  scope,  but  more  accurately  sized  at  the  outset, 
and  consequently  more  effectively  staffed,  should  result  in  project  costs  somewhat  less 
than  the  actual  results  of  NASA's  troublesome  effort 

In  his  work,  Abdel-Hamid  outlines  a  "normalization"  strategy  for  eliminating 
inefficiencies  due  to  initial  project  undersizing  which  incorporates  the  capabilities  of  the 
SD  simulator  Much  of  this  research  work  is  aimed  at  examining  and  testing  this  strategy 
against  the  conventional  calibration  strategy  under  a  variety  of  conditions  and  scenarios. 
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In  theory,  the  normalization  strategy  seeks  to  determine  the  extent  of  man-day 
excesses,  and  adjust  the  archived  calibration/estimation  values  accordingly  Figure  2 
diagrams  both  the  current  calibration  practice  and  the  proposed  normalization  strategy 


(B)  PROPOSED  NORMALIZATION  STRATEGY 

Figure  2.  (a)  Current  Practice:  (b)  Proposed  Normalization  Strategy 
To  determine  the  normalized  cost  value,  a  project  must  be  re-simulated  with  no 
undersizing  Optimization  of  cost  savings  is  determined  by  repeated  simulations  in 
which  actual  project  size  and  schedule  inputs  are  fixed,  while  effort  inputs  are 
systematically  reduced  until  further  input  reductions  begin  to  yield  higher  cost  outputs. 

The  input  and  output  values  generated  during  a  typical  normalization  process  are 
presented  in  Table  11.  Repeated  simulations  in  which  actual  project  effort  (MM(est»  is 
systematically  reduced  with  each  simulation,  yields  a  series  of  actual  costs  (MM(act)). 
The  shaded  cell  in  Table  11  is  the  lowest  numerical  result  generated  by  the  SD  simulator 
under  all  input  conditions.  This  represents  the  project's  "normalized"  man-month  value 
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and  reflects  the  optimum  cost  savings  achievable  in  a  perfectly-sized  project  The 
estimated  versus  actual  cost  values  of  Table  1 1  are  graphically  represented  in  Ftgure  3  to 
further  illustrate  the  normalization  process 


Cycle  #1,  Project  #1 

KDSI  (est) 

TDEV  (est) 

MM  (est) 

MM  (act) 

40 

18.5 

120.9 

120.6 

40 

18.5 

115 

115.3 

40 

18.5 

110 

1 14.6 

40 

18.5 

105 

113.4 

40 

18.5 

100 

112.7 

40 

18.5 

95 

40 

18.5 

90 

112.7 

40 

18.5 

85 

113.3 

40 

18.5 

80 

115.4 

Table  1 1.  Normalization  Values 
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Figure  3.  The  Normalization  Process 
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III.  CONDUCTING  THE  EXPERIMENT 

A.  EXPERIMENTAL  SETTING 

All  experiments  involve  extensive  simulation  modeling  and  cost  estimation  calcula¬ 
tions.  In  addition,  archiving  requirements  for  a  significant  volume  of  generated  data  is 
necessary,  as  well  as  relational  processing  capabilities  to  conduct  comparative  analysis  of 
the  findings.  These  requirements  were  satisfied,  and  the  experimental  tasks  successfully 
accomplished  on  an  IBM-compatible  486-DX2/66  personal  computer  (PC). 

The  System  Dynamics  (SD)  simulator  runs  in  the  MS-DOS  environment,  however  the 
PC  was  configured  to  run  the  application  in  a  window  of  Microsoft  Windows  3.1,  to  fa¬ 
cilitate  transfer  of  information  User  interface  is  via  the  keyboard  Figure  4  is  the 
"changes"  screen,  where  input  parameters  are  entered  to  examine  the  various  exercise 
scenarios.  Of  note,  the  fields  routinely  used  in  experiment  simulations  are  found  on  this 
screen  such  as  DSIPTK  and  UNDEST  (first  column),  TOTMM  (second  column),  and 
TDEV1  (third  column).  A  tailored  report  is  also  generated  for  each  completed  simula¬ 
tion,  and  provides  not  only  a  convenient  presentation  of  simulation  results,  but  also  dis¬ 
plays  initial  input  parameteis  to  permit  easy  verification  of  data  entry.  A  copy  of  one 
such  report  is  presented  in  Figure  5. 

An  electronic  spreadsheet,  specifically  Lotus  1-2-3,  release  4  I  for  Windows,  was  cho¬ 
sen  as  the  appropriate  application  for  managing  and  presenting  the  experimental  data  It 
offers  advanced  spreadsheet,  charting,  drawing,  scenario  and  database  features  which 
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Figure  4.  SD  Simulator  "CHANGES"  Screen 


rtiTXTIAL  ESTIMATES  : 


Project  Size, 
Project  Cost 
Project  Deration 


fixm.  psacsxBerr  results: 


Actual  Project.Size  ■;: 
Completion  Time  - 


24.0  KDSI  ft 

127.3  Person-Months^ 

12.5  Months 


40.0  KDSI 

161.7  Man-Months 

IS. 3  Months 


Figure  5.  Tailored  Simulation  Report 
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were  extremely  valuable  tools  in  conducting,  analyzing,  documenting  and  presenting  the 
results  of  the  experiment 

B.  RELATED  EXPERIMENTAL  TASKS 

With  a  clear  statement  of  the  experimental  objective,  appropriate  choice  of  experi¬ 
mentation  vehicles,  and  a  valid  experiment  design,  several  administrative  tasks  remain 
to  facilitate  conducting  the  experiment  and  handling  the  data.  Important  to  this  pre¬ 
execution  phase  is  the  development  of  a  number  of  worksheet  templates  in  Lotus  1-2-3 
The  "calculations  worksheets"  are  of  particular  value  --  project  profile  data  and  simulated 
project  cost  data  are  directly  entered  here.  Incorporated  within  the  calculations  work¬ 
sheets  are  numeric  cell  formulas  and  interrelationships  such  that  upon  appropriate  entry 
of  project  data,  key  dependent  values  are  automatically  calculated.  Figure  6  is  an  exam¬ 
ple  of  a  calculations  worksheet.  A  detailed  explanation  of  the  calculations  worksheet's 
operation  is  presented  with  the  research  findings  in  Chapter  IV. 

In  addition,  a  number  of  tailored  spreadsheet  tables  were  developed  to  archive,  per¬ 
form  comparative  analysis  on,  and  display  the  collected  data  in  a  consolidated,  readable 
format  Appendix  B  is  an  example  of  this  type  of  tailored  spreadsheet  table. 

C.  DEPENDENT  MEASURES 

Answering  the  research  question  requires  capturing  key  simulation  and  computational 
data  on  project  performance  and  productivity.  These  values  are  absolutely  essential  to 
meaningful  analysis  and  interpretation  of  the  research  findings.  Each  of  these  values  is 
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Figure  6.  Research  Calculations  Worksheet  (Lotus  1-2-3) 
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described  below;  parenthetical  text  following  each  heading  reflects  the  abbreviation  used 

for  this  value  throughout  the  thesis: 

1.  Actual  Project  Effort  (MM(act)) 

Actual  Project  Effort  is  one  of  the  dependent  variables  generated  by  the  SD 
simulator,  and  represents  the  number  of  actual  man-months  required  for  the  software 

development  phase  of  each  individual  project. 

2.  Actual  Project  Schedule  (TDEV(act)) 

This  value  is  also  a  dependent  variable  generated  by  the  SD  simulator,  and  represents 
the  actual  number  of  months  required  for  completion  of  the  software  development  phase 

of  each  individual  project. 

3.  Actual  Project  Productivity  (. Productivity ) 

Actual  Project  Productivity  is  an  important  metric  by  which  competing  calibration 
strategies  are  compared  and  evaluated.  It  is  calculated  by  dividing  the  actual  project  size 
(KDSI(act))  by  the  actual  project  effort  (MM(act)).  This  value  is  calculated  ex-post-facto 
for  each  individual  project.  It  is  expressed  as  a  decimal  value,  and  there  is  an  inverse 

relationship  between  actual  project  effort  and  actual  project  productivity 

4.  Composite  Cycle  Productivity  ( Comp  Prod ) 

Composite  cycle  productivity  is  a  deterministic  value  which  reflects  the  combined 
productivity  of  all  five  projects  as  defined  in  a  particular  project  cycle.  It  is  calculated  by 
dividing  the  total  actual  size  of  all  projects  in  the  cycle  (summation  of  KDSI(act)),  by  the 
total  actual  effort  of  all  projects  (summation  of  MM  (actual)).  Since  the  total  actual  size 
of  all  projects  in  each  cycle  is  fixed  (300  KDSI),  composite  productivity  is  driven  by  the 
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value  of  total  project  effort  -  the  lower  the  total  effort,  the  higher  the  composite 
productivity. 

5.  Average  Staff  (Avg  Staff) 

This  value  represents  the  average  staffing  level  for  each  project.  The  accurate 
projection  of  required  staff  levels  is  a  critical  function  in  software  development. 
Average  Staff  is  calculated  in  COCOMO  by  dividing  the  actual  project  effort  (MM(act)) 

by  the  actual  project  schedule  (TDEV(act)). 

6.  Normalized  Project  Effort  (MM (norm)) 

Normalized  Project  Effort  is  the  value  resulting  from  the  application  of  the 
normalization  process,  described  in  detail  in  Chapter  II,  to  Actual  Project  Effort 
(MM(act)).  Its  value  represents  an  optimal  achievable  level  of  project  effort  and  forms 
the  basis  for  calculation  of  the  COCOMO  Calibration  Coefficient  in  the  alternative 

calibration  strategy  which  is  examined  in  this  experiment. 

7.  COCOMO  Calibration  Coefficient  (Coefficient) 

"Calibration"  is  one  method  by  which  an  organization  may  tailor  a  software  cost 
estimation  tool  to  more  accurately  reflect  its  unique  software  development  experiences. 
"Coefficient"  refers  to  the  constant  term  in  the  COCOMO  nominal  effort  equation,  and 
its  calculated  value  is  critical  to  subsequent  model  estimation  accuracy.  The  central 
issue  in  the  evaluation  of  the  conventional  versus  the  alternate  (normalized)  calibration 
strategies  involves  the  appropriateness  of  the  independent  variable  upon  which  the 
coefficient  calculation  is  based.  In  the  conventional  calibration  strategy,  it  is  based  on 
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actual  project  effort  (MM(act)),  while  the  normalized  calibration  strategy  bases  its 
computation  on  normalized  project  effort  (MM(norm)). 

D.  ORGANIZING  THE  EXPERIMENT 

The  experiment  is  conducted  in  four  phases.  Presented  in  this  section  of  the  report  are 
the  research  objectives  of  the  various  experiments,  an  explanation  of  how  each  phase  is 
organized,  and  a  general  explanation  of  the  exercise  "flow".  Detailed  process  definitions 

are  presented  along  with  the  experimental  results  and  analyses  in  Chapter  IV. 

1.  Phase  One 

The  objective  of  this  phase  is  to  compare  the  simulated  project  cost  results  obtained 
by  applying  the  conventional  software  estimation  tool  calibration  strategy,  against  a 
similar  set  of  cost  values  obtained  by  applying  the  normalized  calibration  strategy.  Both 
learning  and  undersizing  are  assumed  in  this  scenario.  The  project  profile  determines  the 
project-set  order  and  undersizing  allocation  for  each  of  the  six  project  cycles.  The  SD 
simulator  and  COCOMO  equations  are  used  to  both  replicate  the  conventional 
calibration  strategy  and  test  the  alternative  normalization  strategy.  Key  computational 
values  (Dependent  Measures)  are  captured,  and  a  comparative  analysis  of  the  two 
calibration  strategies  is  offered.  The  data  set  collected  in  Phase  One  constitutes  the  "base 

case"  results,  against  which  all  other  scenarios  are  tested. 

2.  Phase  Two 

In  Phases  Two  through  Four,  the  experiment  is  structured  to  perform  sensitivity 
analysis  on  the  base  case  results.  Different  assumptions  and  environmental  factors  are 
examined  by  using  the  SD  simulator's  ability  to  change  one  input  variable  while  holding 
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all  others  constant.  In  each  scenano,  particular  attention  is  paid  to  the  effects  of 
"normalization",  vis-a-vis  the  conventional  calibration  strategy,  on  the  experimental 
results. 

The  objective  of  Phase  Two  is  to  examine  the  effects  of  size  underestimation  on  base 
case  results  A  new  case  is  developed  where  learning  is  assumed,  but  no  size 
underestimation  Simulated  results  for  the  same  project  set  are  calculated,  applying  both 
the  conventional  and  normalized  calibration  strategies,  and  compared  with  base  case 

findings.  All  other  conditions  are  identical  to  those  in  Phase  One. 

3.  Phase  Three 

The  objective  of  Phase  Three  is  to  examine  the  effects  of  learning  on  base  case 
results.  A  new  case  is  developed  where  undersizing  is  assumed,  but  no  learning. 
Simulated  results  for  the  same  project  set  are  calculated,  applying  both  the  conventional 
and  normalized  calibration  strategies,  and  con-pared  with  base  case  findings.  All  other 

conditions  are  identical  to  those  in  Phases  One  and  Two. 

4.  Phase  Four 

The  objective  of  Phase  Four  is  to  examine  the  impact  of  overestimation  and 
underestimation  of  productivity  on  project-set  results.  In  this  scenario,  we  again  assume 
undersizing  and  no  learning,  as  in  the  previous  experiment.  However,  this  experiment 
explores  the  effect  of  misrepresenting  productivity  as  a  function  of  how  the  level  of  effort 
associated  with  the  accomplishment  of  a  software  development  "task"  is  defined  within 
the  organization. 
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Central  to  the  productivity  overestimation/underestimation  question  is  the  notion  of 
"variable  task  definition."  Disparate  definitions  of  the  effort  required  to  accomplish  a 
software  task  may  account  for  situations  where  various  software  development  organiza¬ 
tions  require  different  levels  of  development  effort  to  design  and  code  projects  of  similar 
size  and  scope.  In  projects  where  the  number  of  delivered  source  instructions  is  identical 
in  each  organization,  the  value  of  "task"  becomes  the  determinant  with  regard  to  measur¬ 
ing  effort,  and  hence,  productivity  First,  this  experiment  re-simulates  the  project  set  and 
examines  the  impact  of  underestimating  productivity  by  a  factor  of  75  percent  of  the 
nominal  case  Next,  the  project  set  is  re-simulated,  this  time  overestimating  productivity 
by  a  factor  of  1 25  percent  of  the  nominal  case.  The  results  are  compared  to  Phase  Three, 
which  models  the  nominal  case  in  this  scenario  (undersizing,  no  learning,  and  "perfectly- 
represented"  productivity). 
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IV.  EXPERIMENTAL  RESULTS  AND  ANALYSIS 

A.  INTRODUCTION 

The  SD  simulation  model  generated  raw  data  on  the  actual  cost  and  schedule  for  each 
simulated  project.  The  manner  in  which  these  values  are  applied  in  calibrating  the  CO- 
COMO  software  estimation  tool,  and  its  impact  on  productivity  and  cost  savings  under  a 
series  of  conditions  are  the  central  foc'^s  of  this  analysis.  As  such,  there  are  four  princi¬ 
pal  areas  of  investigation.  First,  the  replication  of  a  conventional  software  estimation 
tool  calibration  strategy  using  raw  cost  data  and  assuming  both  learning  and  undersizing, 
is  compared  with  an  alternative  calibration  strategy  using  normalized  cost  data  under  the 
same  assumptions.  Ne  *  >ase-case  results  of  phase  one  are  compared  with  simulated 
results  of  a  new  case  assuming  learning  but  no  undersizing.  The  third  area  of  investiga¬ 
tion  is  a  comparison  of  the  base-case  results  with  a  new  case  in  which  there  is  undersiz¬ 
ing,  but  without  learning  effects.  Finally,  the  impact  of  both  underestimation  and 
overestimation  of  productivity  on  the  results  obtained  in  the  scenario  with  undersizing 
and  without  learning  is  examined. 
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B.  CONVENTIONAL  VS.  ALTERNATIVE  CALIBRATION  STRATEGIES  WITH 
LEARNING  AND  UNDERSIZING  (BASE  CASE) 

1.  Assumptions 

a.  Underestimation  of  Project  Size 

The  Basic  COCOMO  schedule  estimation  model  requires  as  its  single  input,  a 
user-provided  estimate  of  the  project's  size  in  thousands  of  delivered  source  instructions 
(KDSI).  Consequently,  an  inaccurate  size  estimate  input  will  result  in  a  similarly  impre¬ 
cise  schedule  estimation  output.  The  inclination  toward  project  size  underestimation  is 
not  uncommon  throughout  the  software  industry  (Boehm,  1981,  p  320).  For  purposes  of 
this  experiment,  size  underestimation,  when  applied,  is  represented  as  a  percentage  of 
actual  project  size.  Undersizing  is  assumed  to  range  from  10  percent  to  50  percent,  in 
10-percent  increments,  and  is  applied  to  individual  project  serials  in  accordance  with  the 
project/cycle  profiles  presented  in  Chapter  II.  The  undersizing  percentages,  expressed  in 
decimal  notation,  are  subsequently  applied  as  the  SD  simulator  input  parameter 
UNDEST. 

b.  The  Effects  of  "Learning”  on  Software  Estimation  and  Productivity 

By  "learning"  we  mean  increases  in  productivity.  This  learning  happens  as  an  or¬ 
ganization  gains  experience  in  developing  its  type  of  software  and  as  it  incorporates  new 
software  development  tools.  As  discussed  in  Chapter  II,  we  assume  that  "learning"  is  re¬ 
flected  in  a  10-percent  annual  increase  in  the  SD  simulator  input  parameter  Delivered 
Source  Instructions  per  Task  (DSIPTK).  Consequently,  with  project  schedules  generally 
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approaching  two  yean'  duration,  a  20-percent  increase  in  DSIPTK  was  applied  to  each 
project  cycle  beginning  with  project  cycle  two.  Therefore,  the  learning  scenario  is  de¬ 
fined  as  an  incremental  increase  of  DSIPTK  from  100  percent  to  200  percent  of  the 
nominal  value  over  the  six  project  cycles. 

2.  Conventional  Calibration  Strategy 

Five  synthetic  project  serials  were  simulated  over  six  organizational  project  cycles, 
for  a  total  of  30  simulations.  Key  computational  values,  as  defined  in  Chapter  III,  were 
calculated  and  tracked  throughout  the  experiment.  They  include  Actual  Project  Effort 
|  (MM( actual)).  Actual  Project  Schedule  (TDEV(act)),  COCOMO  Calibration  Coefficient 

i 

(Coefficient),  Actual  Project  Productivity  (Productivity),  Composite  Cycle  Productivity 
(Comp  Prod),  and  Average  Number  of  Staff  Required  (Avg.  Staff).  Appendix  A  presents 
all  calculations  and  data  used  to  generate  these  key  values,  which  are  further  consoli¬ 
dated  and  summarized  in  Table  12. 

!  The  methodology  for  determining  actual  simulated  values  will  be  described  as  the 

i 

! 

process  unfolds  in  Appendix  A.  In  the  following  discussion,  descriptive  abbreviations  in 

I 

parenthesis  correspond  to  column  labels  in  Appendix  A.  For  each  project  serial  (Proj  Se¬ 
rial),  a  learning  value  (DSIPTK  (%))  is  assigned.  A  project  size  estimate  (KDSI(est))  is 
determined  by  multiplying  the  actual  project  size  (KDSl(act))  times  the  size 
<  underestimation  percentage  (Under  (%)).  Using  this  project  size  estimate  (KDSI(est))  as 

the  input  variable  to  the  organic  COCOMO  formula,  the  estimated  project  effort 
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Table  12.  Conventional  Calibration  Strategy 
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(MM(est))  and  estimated  project  schedule  (TDEV(est))  are  determined.  All  required  in¬ 
put  parameters  for  the  project  simulation  have  now  been  calculated.  They  are, 
KDSI(act),  DSIPTK  (%)  —  expressed  as  a  numerical  value  based  on  the  nominal  simula¬ 
tor  value  of  60,  Under  (%)  —  expressed  as  a  decimal  value,  MM(est),  and  TDEV(est). 
Next,  the  SD  simulator  generates  the  actual  effort  (MM(act))  and  actual  schedule 
(TDEV(act))  values. 

The  second  series  of  calculations  presented  in  each  project  cycle  in  Appendix  A, 
uses  the  simulated  actual  effort  and  schedule  values  of  each  of  the  five  project  serials  to 
determine  the  COCOMO  calibration  coefficient  (Coefficient)  which  will  be  applied  to  all 
projects  in  the  subsequent  project  cycle.  Coefficient  calculation  is  based  on  a  series  of 
well-defined  computations  as  described  in  Chapter  II.  In  the  case  of  project  cycle  one, 
the  Coefficient  of  2.56  reflects  an  upward  adjustment  from  the  organic  COCOMO  value 
of  2.4.  If  this  "conventional"  calibration  strategy  is  effective,  this  higher  value,  when  ap¬ 
plied  to  project  cycle  two  size  estimations,  should  produce  more  accurate  effort  and 
schedule  estimates.  Figure  7  shows  the  movement  of  the  COCOMO  calibration  coeffi¬ 
cient  over  the  six  project  cycles  under  the  conventional  calibration  strategy. 

In  addition,  actual  project  productivity  (Productivity)  and  composite  cycle  productiv¬ 
ity  (Comp  Prod)  are  also  determined  in  Appendix  A.  Actual  project  productivity  (Pro¬ 
ductivity)  is  defined  as  the  actual  size  of  the  project  (KDSI(act))  divided  by  the  actual 
cost  of  the  project  (MM(actual)).  Results  of  the  experiment  are  displayed  in  Figure  8, 
and  reflect  individual  project  productivities  between  .27  and  .43. 


45 


COCOMO  Calibration  Coefficient 

(Conventional  Calibration  Strategy  -  Base  Case) 


Productivity:  Conventional  Calibration  Strategy  -  Base  Case 


Composite  cycle  productivity  is  defined  as  the  total  actual  size  of  all  projects  in  the 

cycle  (XKDSI  (act)),  divided  by  the  total  actual  effort  of  all  projects  (£mm  (act)).  In 
die  conventional  calibration  scenario,  overall  composite  productivity  of  the  software  de¬ 
velopment  organization  through  the  six  project  cycles  improved  from  .317  to  .41 1  (29.65 
percent).  Figure  9  captures  this  upward  movement  of  composite  productivity. 

3.  Alternative  Calibration  Strategy 

The  methodology  employed  in  applying  the  alternative  calibration  strategy  is  identi¬ 
cal  to  the  conventional  strategy  described  in  die  previous  section,  with  one  important  ex¬ 
ception.  As  described  in  Chapter  n,  upon  determination  of  actual  cost  and  schedule 
values  using  conventional  COCOMO  techniques,  the  projects  are  re-simulated  with  ac¬ 
tual  size  and  actual  schedule  inputs  fixed.  Cost  estimates  are  gradually  reduced  from  the 
actual  simulated  value  until  die  optimum  savings,  or  "normalized''  cost  value,  is 
achieved.  Appendix  B  provides  all  data  on  the  normalization  process  for  each  of  die  five 
project  serials  over  the  six  project  cycles.  Shaded  cells  in  the  MM(act)  column  represent 
the  optimum  or  "normalized”  value  for  that  particular  project  This  value,  referred  to  as 
MM(norm),  is  incorporated  in  the  organizational  data  base  and  is  used  to  calculate  the 
new  COCOMO  calibration  coefficient  Appendix  C  presents  all  calculations  and  data  as¬ 
sociated  with  the  calibration  of  COCOMO  using  normalized  data.  Note  its  similarities 
with  Appendix  A  However,  in  die  second  series  of  calculations  for  each  project  cycle, 
the  normalized  effort  (MM(norm))  is  a  new  column  entry.  Its  value  was  computed  as 
part  of  the  normalization  process  and  transferred  directly  from  the  shaded  cells  in 
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Composite  Productivity:  Conventional  Calibration  Strategy  -  Base  Case 


Appendix  B.  It  is  this  value,  MM(norm),  which  generates  the  new  COCOMO  coeffi¬ 
cient,  and  not  the  actual  effort  cost  value  (MM(act»,  as  in  the  conventional  calibration 
strategy. 

It  is  important  to  note  that  normalization  of  the  effort  cost  data  has  no  direct  impact 
on  project  productivity  or  composite  cycle  productivity,  as  actual  effort  costs  continue  to 
be  used  in  computing  these  values.  Normalization  is  primarily  a  process  by  which  the  in¬ 
efficiencies  which  have  plagued  a  problematic  software  development  project  can  be 
eliminated.  In  so  doing,  it  is  possible  for  an  organization  to  optimize  the  accuracy  and 
representativeness  of  archived  data  for  future  estimation  of  similar  projects. 

A  by-product  of  the  normalization  process,  however,  is  improved  productivity.  In 
theory,  normalization  provides  the  organization  with  more  optimal  calibration  coeffi¬ 
cients  which  should  lead  to  more  optimal  estimations.  As  inefficiencies  are  eliminated  in 
project  estimation,  simulations  produce  projects  with  lower  actual  costs,  which  in  turn, 
lead  to  improved  productivity.  These  notions  are  borne  out  in  the  experimental  findings 
summarized  in  Figure  10  and  Table  13  —  a  comparison  of  the  previously-determined  raw 
historical  data  with  the  normalized  data  recorded  upon  re-simulation  of  the  identical  pro¬ 
ject  set  Improvement  percentages  for  normalized  data  versus  raw  data  are  calculated  in 
Table  13  for  actual  cost  productivity,  and  composite  productivity  values.  Note  that  be¬ 
ginning  with  project  cycle  two  (when  the  normalization  process  first  produces  a  unique 
calibration  coefficient),  improvement  is  noted  across  ail  entries.  While  improvements 
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Composite  Productivity  -  Base  Case 


Table  13.  Comparison  of  Conventional  and  Normalized  Calibration  Strategies  -  Base  Case 
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Cycle  &  Project  Raw  Data  Normalized  Data  Percent  Improvement 


are  noted  in  productivity  values  associated  with  both  raw  and  normalized  data,  the  more 
dramatic  results  achieved  through  data  normalization  is  apparent. 

Of  particular  significance  is  the  improvement  in  composite  cycle  productivity  evident 
within  both  the  raw  and  normalized  data  sets  themselves.  Over  the  course  of  the  six  pro¬ 
ject  cycles,  composite  productivity',  as  determined  under  the  conventional  calibration 
strategy  improved  by  29.65  percent  (from  .317  to  .41 1).  Even  more  impressively,  under 
the  normalization  strategy,  composite  productivity  values  improved  by  42.27  percent 
(from  .317  to  .451).  Recalling  that  in  this  scenario,  experimental  assumptions  include 
both  learning  and  undersizing,  it  is  logical  to  pursue  investigation  of  alternative  scenarios 
in  an  effort  to  isolate  and  examine  the  effects  of  these  assumptions. 

The  proper  use  of  normalized  effort  cost  data  can  have  a  significant  impact  on  future 
software  development  costs.  Table  14  summarizes  actual  project  effort  (MM(act))  under 
both  the  conventional  and  normalized  calibration  strategies.  In  addition,  the  table  in¬ 
cludes  information  on  potential  savings  which  may  be  achieved  by  archiving  normalized 
data  in  the  organizational  data  base  vice  the  actual  cost  data.  These  savings  could  result 
when,  in  the  future,  the  organization  is  faced  with  estimation  of  a  project  of  similar  size 
and  scope.  By  using  normalized  data  as  input,  estimates  would  not  be  biased  by  the  inef¬ 
ficiencies  which  plagued  the  previous  project.  The  potential  savings  in  our  problem  set 
are  noteworthy,  both  in  terms  of  real  effort  cost  savings  (2.2  to  34.4  man-months)  and 
percentage  of  reduction  in  cost  (1.05  to  16.92  percent).  Figure  1 1  graphically  represents 
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Cycle  and  Project  Information 

Actual  Project  Effort 

Potential  Savings  Through 

1 

Conventional 

Normalized 

Norr^tization 

1 

Percent 


0.00% 


0.00% 


0.00% 


0.00% 


0.00% 


3.12% 


8.58% 


7.11% 


4.63% 


1.05% 


15.89% 


12.90% 


15.03% 


16.53% 


8.48% 


11.75% 


11.95% 


10.22% 


2.16% 


11.32% 


Table  14.  Potential  Savings  Through  Normalization 
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Potential  Cost  Savings  Through  Normalization 


Figure  11.  Potential  Cost  Savings  Achievable  Through  Normalization  (Man-Months) 


the  potential  cost  savings  achievable  through  normalization  of  all  projects,  beginning 
with  project  cycle  two. 

These  savings  are  possible  since  normalization  removes  the  inefficiencies  which  lead 
to  smaller  COCOMO  coefficients,  which  in  turn,  lead  to  "tighter"  (i.e.,  smaller)  cost  esti¬ 
mates.  On  the  other  hand,  the  conventional  calibration  strategy  produces  higher  calibra¬ 
tion  coefficients  which  subsequently  lead  to  larger  size  estimates  (Figure  12).  As 
discussed  in  Chapter  II,  these  higher-than-ideal  estimates  significantly  influence  the  pro¬ 
ject's  final  results.  Work  expands  to  fill  the  available  slack  time,  and  the  self-fulfilling 
prophecy  of  Parkinson's  Law  is  realized  once  again  (Boehm,  1981,  p.  592). 

Estimated  project  productivity  was  calculated  as  a  measure  by  which  the  effects  of 
project  size  underestimation  could  be  observed  on  project  behavior  and  outcome.  Its  cal¬ 
culation  differs  from  that  of  actual  productivity  in  that  the  actual  size  of  the  project 
(KDSI(act))  is  divided  by  the  COCOMO-generated  estimate  of  project  cost  based  on  no 
size  underestimation  (MM(est)).  With  post-facto  knowledge  of  a  project's  actual  size,  an 
estimated  project  effort  value  can  be  generated  for  the  denominator  value  (MM(est)). 
Figure  13  plots  estimated  project  productivity  versus  project  size  for  project  cycle  one 
and  both  the  conventional  and  normalized  estimated  productivity  values  for  project  cycle 
six.  It  is  clear  from  the  plot  that  estimated  productivity  decreases  as  project  size  increases 
in  all  three  instances. 

As  defined,  the  estimated  productivity  value  should  "shadow”  the  actual  productivity 
value  as  it  relates  to  the  COCOMO-calibrated  project  effort  estimate.  When  compared 
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against  actual  project  productivity,  estimated  project  productivity  provides  an  indication 
of  the  relative  accuracy  and  validity  of  the  software  estimation  tool  and  its  calibration  co¬ 
efficient.  Figures  14,  15,  and  16  compare  actual  versus  estimated  project  productivity  as 
a  function  of  project  size  for  project  cycles  one  and  both  the  raw  and  normalized  in¬ 
stances  of  project  cycle  six,  respectively.  In  Figure  14,  the  trend  toward  convergence  of 
the  actual  and  estimated  productivity  values  appears  loosely  related  to  initial  project  un¬ 
dersizing.  For  example,  the  project  with  the  smallest  size  underestimation  (80  KDSI  with 
10%  underestimation)  has  an  actual  productivity  figure  closest  to  its  estimated  productiv¬ 
ity  value.  Likewise,  the  actual  productivity  of  the  project  with  the  largest  undersizing  (70 
KDSI  with  50%  underestimation)  is  furthest  away  from  its  estimated  counterpart. 

From  Figure  13,  it  is  evident  that  the  conventional  COCOMO  calibration  method  has 
lead  to  estimated  productivity  values  in  project  cycle  six  approximately  10  percent  more 
than  similar  projects  in  cycle  one.  The  normalization  method  yields  values  nearly  41  per¬ 
cent  higher  than  cycle  one.  Nevertheless,  from  Figure  15,  conventional  cycle  six  actual 
productivity  values  exceeded  their  estimates  by  between  5.1  and  14  percent.  With  the 
exception  of  the  largely  undersized  project  (70  KDSI,  50-percent  undersizing),  the  nor¬ 
malization  strategy,  shown  in  Figure  16,  provides  the  best  "fit",  with  estimated  produc¬ 
tivities  exceeding  actual  productivities  by  an  average  of  less  than  1.5  percent. 

This  fact  is  also  confirmed  by  using  the  completed  project  results  for  ex-post-facto 
evaluation  of  the  accuracy  of  the  COCOMO  estimation  model.  The  percentage  of 
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relative  error  in  the  accuracy  of  project  cost  estimation  can  be  caluclated  using  the  fol¬ 


lowing  equation: 


Percent  Relative  Error  = 


1 00*  \MM(act)-MM(est)  I 


(4.1) 


MM(act) 

Equation  4. 1  is  used  to  determine  the  accuracy  of  the  base  case  estimates  generated  under 
both  the  conventional  and  normalized  calibration  strategies  in  cycles  two  through  six  of 
the  exercise  scenario.  Figure  17  is  a  plot  of  the  average  error  for  all  projects  by  project 
cycle,  and  the  results  suggest  that  the  accuracy  of  COCOMO  project  cost  estimation  in 
this  scenario  favors  the  normalized  calibration  model  over  the  conventional  model. 


C.  EFFECTS  OF  NO  UNDERSIZING  ON  BASE  CASE  RESULTS 


Having  concluded  an  examination  of  conventional  versus  normalized  calibration 
strategies  in  a  scenario  that  included  both  learning  and  undersizing  (base  case),  the  pro¬ 
ject  set  was  re-simulated  under  similar  conditions,  but  assuming  no  undersizing.  The 
methodology  was  identical  to  the  base  case,  with  the  exception  that  the  SD  simulator  in¬ 
put  UNDEST  was  set  at  "0"  in  each  project  simulation  to  reflect  "perfect"  size  estimation. 
Appendices  D,  E,  and  F  document  the  results  of  these  re-simulations,  again  modeling 
both  the  conventional  and  normalized  calibration  strategies.  The  results  are  summarized 
in  Table  15. 

A  comparison  with  the  base  case  results  (Table  13)  reveals  some  interesting  findings. 
With  no  undersizing,  individual  productivity  improved  in  all  projects  and  across  all  pro¬ 
ject  cycles  with  respect  to  their  undersized  counterparts.  In  1 8  of  the  30  project  serials, 
however,  the  percentage  of  improvement  in  productivity  realized  through  the 
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Table  15.  Comparison  of  Conventional  and  Normalized  Calibration  Strategies: 
Case  With  Learning  and  No  Undersizing 


normalization  process,  was  less  in  this  scenario  (no  undersizing)  than  in  the  base  case 
(with  undersizing).  This  is  reflected  in  Figure  18,  where  a  plot  of  the  average  improve¬ 
ment  in  productivities  as  a  result  of  normalization  shows  minimal  variance  between  the 
two  scenarios. 

Composite  cycle  productivities  within  the  domain  of  the  "no  undersizing"  scenario, 
again  showed  a  significant  improvement  over  the  span  of  the  six  project  cycles,  with  the 
conventional  strategy  yielding  an  improvement  of  32.3  percent,  and  the  normalization 
strategy  43.7  percent.  These  productivity  improvements  (without  undersizing),  however, 
are  only  marginally  better  than  those  realized  in  the  base  case  (with  undersizing).  Figure 
19  presents  a  graphical  summary  of  composite  cycle  productivity,  comparing  raw  and 
normalized  results  in  both  tire  undersizing  and  no-undersizmg  scenario.  It  is  evident  that 
by  the  third  project  cycle,  composite  productivity  under  the  normalized  calibration  strat¬ 
egy  surpasses  the  productivity  values  achieved  under  the  conventional  calibration  strat¬ 
egy,  regardless  of  whether  or  not  the  project's  size  was  underestimated.  This  finding 
suggests  that  normalization  may  be  an  effective  tool  that  can  help  offset  the  negative  ef¬ 
fects  of  project  estimation  undersizing.  Nevertheless,  further  research  is  required  to  sup¬ 
port  this  claim. 

Estimated  productivity  comparisons  under  this  scenario  reveal  some  interesting  re¬ 
sults.  With  no  undersizing,  actual  and  estimated  individual  project  productivities  are 
nearly  identical  in  cycle  one  (Figure  20).  These  values  are  the  same  in  the  conventional 
and  normalized  cases,  since  the  initial  effect  of  the  normalization  process  is  not  evident 
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Average  Improvement  in  Productivity  Through  Normalization 
(Base  Case  vs  Case  With  Learning  And  No  Undersizing) _ 


68 


until  project  cycle  two.  However,  using  the  conventional  strategy  (raw  data),  estimates 
of  productivity  begin  to  drift,  and  by  cycle  six  lag  actual  productivities  by  a  range  of  6.8 
percent  to  10.86  percent  (Figure  21).  Conversely,  normalized  data  continues  to  produce 
precise  estimates  within  one  percent  of  actual  productivity  values  in  cycle  six  (Figure 
22).  This  would  indicate  a  more  responsive  calibration  of  the  COCOMO  constant  by  the 
normalization  process  in  this  scenario. 

The  relative  error  in  the  accuracy  of  COCOMCs  project  cost  estimation  under  con¬ 
ventional  and  normalized  calibration  strategies  is  quite  dramatic  in  this  scenario  of  no 
undersizing,  as  can  be  clearly  seen  in  Figure  23.  With  "perfect"  size  input,  normalization 
of  the  data  results  in  consistent  COCOMO  cost  estimates  across  all  project  cycles,  with  a 
relative  error  rate  of  less  than  one-half  percent  Conversely,  while  conventionally- 
calibrated  COCOMO  produces  "tight"  cost  estimates  in  project  cycles  one  and  two,  the 
error  rate  balloons  to  nearly  ten  percent  by  cycle  six. 

D.  EFFECTS  OF  NO  LEARNING  ON  BASE  CASE  RESULTS 

In  this  experiment  the  project  set  was  re-simulated  in  a  scenario  which  included  un¬ 
dersizing,  but  assumed  no  learning  between  project  cycles.  The  methodology  differed 
from  the  base  case  only  in  the  fact  that  the  SD  simulator  parameter  DSIPTK  remained 
fixed  at  the  default  value  of  "60"  for  all  project  simulations.  This  effectively  eliminated 
the  learning  assumption,  by  modeling  the  experiment  with  a  "flat"  delivered-source- 
mstruction-per-task  rate  from  cycle  to  cycle.  Appendices  G,  H,  and  I  document  the  results 
of  the  experiment. 
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Results  of  the  experiment  are  summarized  in  Table  16,  and  show  that  while  individ¬ 
ual  project  productivity  using  the  conventional  calibration  strategy  varied  between  .26 
and  .35,  composite  productivity  through  the  six  project  cycles  decreased  marginally  from 
.317  to  .31 1  (1.89  percent).  In  this  scenario  (undersizing  but  no  learning),  the  normali¬ 
zation  strategy  yielded  minimal  improvement,  at  best,  over  the  conventional  strategy  in 
terms  of  real  effort  (-2.92  percent  to  6.54  percent),  individual  project  productivity  (-3.85 
percent  to  6.25  percent)  and  comf  site  productivity  (.33  percent  to  3.57  percent).  In  ad¬ 
dition,  with  normalization,  composite  productivity  over  the  six  project  cycles  improved 
only  trivially  from  .317  to  .318  (.315  percent).  These  composite  productivity  values  are 
graphically  represented  in  Figure  24,  and  provide  an  important  observation.  The  findings 
suggest  that,  in  an  environment  devoid  of  learning,  both  the  conventional  and  normaliza¬ 
tion  calibration  strategies  are  largely  ineffective  in  improving  productivity. 

Similarly,  both  estimated  productivity  and  relative  accuracy  values  are  inconclusive 
in  this  scenario.  In  the  case  of  the  conventional  strategy,  raw  data  values  produce  under¬ 
estimates  of  productivity  averaging  4.5  percent,  while  the  normalization  strategy  yields 
overestimates  averaging  8.9  percent.  The  accuracy  of  project  cost  estimation  favors  the 
conventional  COCOMO  calibration  strategy  in  three  of  the  five  project  serials,  besting 
the  normalized  model's  average  relative  error  rate,  6.08  percent  to  7.62  percent. 
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Case  With  Undersizing  and  No  Learning 


E.  THE  EFFECTS  OF  OVERESTIMATION  AND  UNDERESTIMATION  OF 
PRODUCTIVITY  ON  SIMULATION  RESULTS 

The  final  series  of  experiments  examines  the  impact  of  overestimation  /  underestima¬ 
tion  of  productivity  on  project  set  results.  In  this  scenario,  we  again  assume  undersizing 
and  no  learning,  as  in  the  previous  experiment.  However,  this  experiment  explores  the 
effect  of  misrepresenting  productivity  by  virtue  of  how  a  "task"  is  defined. 

Central  to  the  notion  of  variable  task  definition  is  the  situation  where  different  soft¬ 
ware  development  organizations  require  different  development  efforts  to  design  and  code 
projects  of  a  similar  size  and  ,  scope.  Consequently,  where  DSI  is  constant  and  fixed  in 
both  organizations,  the  value  of  "task"  becomes  the  determinant  with  regard  to  measuring 
effort. 

First,  the  project  set  is  re-simulated  with  underestimation  and  no  learning,  but  with  a 
DSIPTK  value  fixed  at  75  percent  of  the  nominal  case.  The  nominal  case  default  value 
of  the  SD  simulator  is  "60",  hence,  the  input  metric  is  set  at  "45".  Cost  and  productivity 
values  are  calculated  in  the  usual  manner,  using  both  the  conventional  and  normalization 
calibration  strategies.  Data  and  calculations  are  presented  in  Appendices  J,  K,  and  L,  and 
are  summarized  in  Table  17.  A  comparison  with  Table  16  values  (undersizing,  no  learn¬ 
ing,  nominal  DSIPTK  value),  and  employing  the  conventional  strategy  with  raw  histori¬ 
cal  data,  reveals  significantly  lower  individual  project  productivities  in  each  instance. 
Likewise,  composite  cycle  productivities  fall  by  15.5  percent  to  17.8  percent.  The  effects 
of  normalization  under  these  experimental  conditions  are  negligible.  Both  individual 
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Table  17.  Comparison  of  Conventional  and  Normalized  Calibration  Strategies: 

Case  With  Undersizing,  No  Learning  and  DSIPTK  =  75%  of  Nominal  Case 
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Project  Raw  Data:  DSIPTK  ■=  75%  Normalized:  DSIPTK  *  75%  j  Percent  Improvement 


project  productivities  and  composite  cycle  productivities  are  virtually  unchanged  despite 
normalization  (improvement  range  of  -.38  percent  to  2.37  percent). 

Next,  the  DSEPTK  value  was  set  at  125  percent  of  die  nominal  case,  or  "75",  and  the 
projects  re-simulated  yet  again  with  all  other  conditions  unchanged  Supporting  data  and 
calculations  are  presented  in  Appendices  M,  N,  and  O,  and  are  summarized  in  Table  18. 
Results  under  the  conventional  strategy  reveal  a  global  improvement  in  individual  project 
productivity.  Similarly,  composite  cycle  productivity  improves  by  an  average  of  10.34 
percent  over  Table  16  (nominal)  values.  The  effect  of  normalization  in  this  scenario, 
while  not  as  dramatic  as  under  the  learning  assumption  (Table  13),  nevertheless  improves 
composite  productivity  by  an  average  of  1 1.96  percent  over  the  Table  16  values,  and 
yields  an  improvement  over  conventional  strategy  values  ranging  from  2.09  to  4.85 
percent. 

Figure  25  is  a  graphical  representation  of  composite  productivity  under  all  exercise 
conditions  described  in  this  section,  and  includes  data  carried  forward  from  the  previous 
section  (DSIPTK  =  100%)  for  comparison  purposes.  The  composite  productivity  posi¬ 
tioning  is  readily  apparent  and  appears  directly  linked  to  DSIPTK  values'percentages. 
The  figure  also  provides  a  view  of  the  effects  of  normalization  on  each  of  the  three  data 
sets.  Clearly,  the  higher  DSIPTK  values  yield  the  more  significant  normalization  benefit 
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Composite  Productivity  -  Conventional  vs  Normalized 

(Comparison  of  Results  Based  on  DSIPTK  =  75%,  100%  and  125%) 
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Figure  25.  Composite  Productivity:  Conventional  vs  Normalized 
DSIPTK  =  75%,  100%,  and  125% 


V.  CONCLUSIONS 


A.  SUMMARY  OF  FINDINGS  AND  IMPLICATIONS 

The  major  objective  of  this  thesis  was  to  use  simulation  modeling  to  replicate  the  de¬ 
velopment  of  a  set  of  30  hypothetical  software  projects,  the  results  of  which  were  used  to 
evaluate  two  competing  calibration  strategies  for  the  COCOMO  software  estimation  tool 

in  four  experimental  scenarios. 

1.  Phase  One 

In  phase  one,  the  simulated  project  costs  obtained  by  applying  the  conventional 
calibration  strategy,  were  evaluated  against  a  similar  set  of  cost  values  obtained  by 
applying  the  normalized  calibration  strategy  in  a  scenario  which  assumed  both  learning 
and  undersizmg  The  normalization  process  contributed  to  significant  increases  in  both 
individual  project  productivity  and  composite  cycle  productivity.  The  experiment 
demonstrated  that  normalization  provided  the  organization  with  more  optimal  calibration 
coefficients  which,  in  turn,  lead  to  more  optimal  cost  estimations.  As  inefficiencies  were 
eliminated  in  project  cost  estimation,  simulations  produced  projects  with  lower  actual 
costs,  and  hence,  improved  productivity. 

The  experiment  also  demonstrated  that  the  normalization  strategy  provided  the  soft¬ 
ware  organization  with  the  potential  for  significant  future  cost  savings.  The  normaliza¬ 
tion  process  effectively  removed  many  of  the  inefficiencies  associated  with  undersized 
projects.  Consequently,  archiving  normalized  cost  data  in  the  organizational  data  base 
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vice  the  actual  project  results,  produced  more  optimal  estimates  when  identical  projects 
were  re-simulated  following  model  calibration.  In  contrast,  as  a  result  of  higher  calibra¬ 
tion  coefficients,  the  conventional  calibration  strategy  produced  consistently  larger  and 
less  optimal  cost  estimates. 

Post-facto  knowledge  of  the  projects'  actual  size  was  used  to  calculate  two  related  ex¬ 
ercise  metrics,  both  of  which  provided  an  indication  of  the  relative  accuracy  and  validity 
of  the  software  estimation  tool  —  estimated  productivity  and  relative  error  in  cost  estima¬ 
tion.  The  normalized  cost  data  produced  the  strongest  correlation  between  actual  and  es¬ 
timated  productivity  results,  indicating  that  the  model  provided  more  accurate  estimates. 
This  was  confirmed  when  the  computed  accuracy  of  the  base  case  COCOMO  estimates 

clearly  favored  the  normalized  calibration  model. 

2.  Phase  Two 

In  phase  two,  the  base  case  results  of  phase  one  were  compared  with  simulated 
results  of  a  new  case  assuming  learning,  but  no  undersizing.  With  no  undersizing,  both 
the  conventional  and  normalized  calibration  strategies  produced  global  improvements  in 
project  productivities  over  base  case  results.  Normalization  again  provided  cost  benefit 
over  raw  historical  data,  but  in  this  scenario,  the  average  improvement  in  individual 
project  productivity  was  less  dramatic  than  in  the  base  case.  Similarly,  composite  cycle 
productivities  were  only  marginally  improved  over  their  base  case  counterparts.  These 
findings  suggest  that  normalization  may  be  an  effective  strategy  to  counterbalance  the 
detrimental  effects  of  initial  project  undersizing.  Both  estimated  productivity  and 
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relative  accuracy  solutions  in  this  scenario  revealed  that  the  conventional  calibration 
strategy  produced  increasingly  suboptimal  model  performance  over  the  six  project  cycles 
Conversely,  the  normalized  model  continued  to  provide  extremely  precise  estimates 
throughout  all  project  cycles. 

3.  Phase  Three 

Phase  three  re-simulated  the  project  set  in  a  scenario  which  included  project  size 
underestimation,  but  no  learning.  Normalization  was  least  effective  in  this  scenario, 
yielding  minimal  improvement,  at  best,  over  the  conventional  strategy  in  all  key  cost  and 
productivity  metrics.  The  findings  suggest  that  without  learning,  both  the  conventional 
and  normalization  calibration  strategies  are  largely  ineffective  in  improving  productivity. 

A  comparison  of  relative  model  accuracy  was  also  inconclusive  in  this  scenario. 

4.  Phase  Four 

The  final  phase  of  the  experiment  investigated  the  impact  of  both  underestimation  and 
overestimation  of  productivity  on  the  results  of  the  phase  three  experiment.  First,  with 
productivity  underestimated  by  a  factor  of  75  percent  of  the  nominal  case,  all  productiv¬ 
ity  metrics  were  degraded,  and  normalization  had  a  negligible  impact.  Next,  with  produc¬ 
tivity  overestimated  by  a  factor  of  125  percent  of  the  nominal  case,  all  productivity 
values  showed  improvement  Normalization  was  again  effective  in  this  scenario,  but  less 
dramatically  than  in  the  base  case  (learning  and  no  undersizing).  Productivity  in  this  sce¬ 
nario  appears  directly  linked  to  the  concept  of  variable  task  definition  as  it  relates  to  the 
number  of  delivered  source  instructions  per  task  (DSIPTK).  In  addition,  the  effects  of 
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normalization  also  tend  to  follow  this  DSIPTK  movement  -  the  higher  DSIPTK  values 
yield  the  more  significant  normalization  benefit. 

B.  FURTHER  RESEARCH  RECOMMENDATIONS 

Three  interesting  research  directions  could  be  pursued  as  follow-on  to  this  study.  The 

first  possibility  is  a  validation  of  the  findings  of  this  simulation-based  study  by  conduct¬ 
ing  an  experiment  in  a  real  organization  to  compare  the  two  strategies.  Second,  the  cur¬ 
rent  normalization  strategy  seeks  to  eliminate  the  inefficiencies  caused  by  underlining 
The  SD  simulator  could  be  used  to  examine  the  possibilities  of  eliminating  other  sources 
of  inefficiency  such  as  the  mi  sal  location  of  staff  resources.  Third  the  normalization 
process  requires  repeated  simulations  to  arrive  at  the  optimal  cost  solution,  and  as  such, 
is  quite  labor  and  time-intensive.  The  possibility  for  automating  the  process,  perhaps  em¬ 
ploying  artificial  intelligence  techniques,  could  be  investigated. 
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APPENDIX  A.  CONVENTIONAL  CALIBRATION  STRATEGY: 

BASE  CASE 
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APPENDIX  C.  NORMALIZATION  CALIBRATION  STRATEGY: 

BASE  CASE 
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APPENDIX  D.  CONVENTIONAL  CALIBRATION  STRATEGY: 
LEARNING  -  NO  UNDERSIZING 
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CYCLE  #5  (Raw  Data.  180%  DSIPTK  NO  UNDERSIZING) 
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CYCLE  #6  (Raw  Data.  200%  DSIPTK.  NO  UNDERSIZING) 
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APPENDIX  E.  NORMALIZATION  CALIBRATION  STRATEGY: 
LEARNING  -  NO  UNDERSIZING 
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CYCLE  #6  (SofTT»{o*d  Data  ISO*  OSfPTK  NO  UNDERSfZING)) 
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APPENDIX  F.  NORMALIZATION  DATA 
LEARNING  -  NO  UNDERSIZING 


CYCLE  #1,  PROJECT  #1 
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APPENDIX  G.  CONVENTIONAL  CALIBRATION  STRATEGY 
LNDERSIZING  -  NO  LEARNING 
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APPENDIX  H.  NORMALIZATION  CALIBRATION  STRATEGY: 
UNDERSIZING  -  NO  LEARNING 
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APPENDIX  I.  NORMALIZATION  DATA: 
UNDERSIZING  -  NO  LEARNING 
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APPENDIX  J.  CONVENTIONAL  CALIBRATION  STRATEGY 
UNDERSIZING  -  75%  DSIPTK 
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